Expanded Access to Video-Based Laparoscopic Skills Assessments: Ease, Reliability, and Accuracy.

OBJECTIVE: Video-based performance assessments provide essential feedback to surgical residents, but in-person and remote video-based assessment by trained proctors incurs significant cost. We aimed to determine the reliability, accuracy, and difficulty of untrained attending staff surgeon raters completing video-based assessments of a basic laparoscopic skill. Secondarily, we aimed to compare reliability and accuracy between 2 different types of assessment tools. DESIGN: An anonymous survey was distributed electronically to surgical attendings via a national organizational listserv. Survey items included demographics, rating of video-based assessment experience (1 = have never completed video-based assessments, 5 = often complete video-based assessments), and rating of favorability toward video-based and in-person assessments (0 = not favorable, 100 = favorable). Participants watched 2 laparoscopic peg transfer performances, then rated each performance using an Objective Structured Assessment of Technical Skill (OSATS) form and the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS). Participants then rated assessment completion ease (1 = Very Easy, 5 = Very Difficult). SETTING: National survey of practicing surgeons. PARTICIPANTS: Sixty-one surgery attendings with experience in laparoscopic surgery from 10 institutions participated as untrained raters. Six experienced laparoscopic skills proctors participated as expert raters. RESULTS: Inter-rater reliability was substantial for both OSATS (k = 0.75) and MISTELS (k = 0.85). MISTELS accuracy was significantly higher than that of OSATS (κ: MISTELS = 0.18, 95%CI = [0.06,0.29]; OSATS = 0.02, 95%CI = [-0.01,0.04]). While participants were inexperienced with completing video-based assessments (median = 1/5), they perceived video-based assessments favorably (mean = 73.4) and felt assessment completion was "Easy" on average. CONCLUSIONS: We demonstrate that faculty raters untrained in simulation-based assessments can successfully complete video-based assessments of basic laparoscopic skills with substantial inter-rater reliability without marked difficulty. These findings suggest an opportunity to increase access to feedback for trainees using video-based assessment of fundamental skills in laparoscopic surgery.

Is There a Problem With Evidence in Health Professions Education?

ABSTRACT: What constitutes evidence, what value evidence has, and how the needs of knowledge producers and those who consume the knowledge produced as evidence might be better aligned are questions that continue to challenge the health sciences. In health professions education (HPE), debates on these questions have ebbed and flowed with little sense of resolution or progress. In this article, the authors explore whether there is a problem with evidence in HPE using thought experiments anchored in Argyris' learning loops framework.From a single-loop perspective ("How are we doing?"), there may be many problems with evidence in HPE, but little is known about how research evidence is being used in practice and policy. A double-loop perspective ("Could we do better?") suggests expectations of knowledge producers and knowledge consumers might be too high, which suggests more systemwide approaches to evidence-informed practice in HPE are needed. A triple-loop perspective ("Are we asking the right questions?") highlights misalignments between the dynamics of research and decision-making, such that scholarly inquiry may be better approached as a way of advancing broader conversations, rather than contributing to specific decision-making processes.The authors ask knowledge producers and consumers to be more attentive to the translation from knowledge to evidence. They also argue for more systematic tracking and audit of how research knowledge is used as evidence. Given that research does not always have to serve practical purposes or address the problems of a particular program or institution, the relationship between knowledge and evidence should be understood in terms of changing conversations, as well as influencing decisions.

Identifying Contact Time Required for Secondary Transmission of Clostridioides difficile Infections by Using Real-Time Locating System.

Considering patient room shortages and prevalence of other communicable diseases, reassessing the isolation of patients with Clostridioides difficile infection (CDI) is imperative. We conducted a retrospective study to investigate the secondary CDI transmission rate in a hospital in South Korea, where patients with CDI were not isolated. Using data from a real-time locating system and electronic medical records, we investigated patients who had both direct and indirect contact with CDI index patients. The primary outcome was secondary CDI transmission, identified by whole-genome sequencing. Among 909 direct and 2,711 indirect contact cases, 2 instances of secondary transmission were observed (2 [0.05%] of 3,620 cases), 1 transmission via direct contact and 1 via environmental sources. A low level of direct contact (113 minutes) was required for secondary CDI transmission. Our findings support the adoption of exhaustive standard preventive measures, including environmental decontamination, rather than contact isolation of CDI patients in nonoutbreak settings.

National Profile of the ACGME Milestones 1.0 and 2.0 within General Surgery: A Seven-Year National Study from 2014 to 2021.

PURPOSE: The Accreditation Council for Graduate Medical Education (ACGME) introduced General Surgery Milestones 1.0 in 2014 and Milestones 2.0 in 2020 as steps toward competency-based training. Analysis will inform residency programs on curriculum development, assessment, feedback, and faculty development. This study describes the distributions and trends for Milestones 1.0 and 2.0 ratings and proportion of residents not achieving the level 4.0 graduation target. METHODS: A deidentified dataset of milestone ratings for all ACGME-accredited General Surgery residency programs in the United States was used. Medians and interquartile ranges (IQR) were reported for milestone ratings at each PGY level. Percentages of PGY-5s receiving final year ratings of less than 4.0 were calculated. Wilcoxon rank sum tests were used to compare 1.0 and 2.0 median ratings. Kruskal-Wallis and Bonferroni post-hoc tests were used to compare median ratings across time periods and PGY levels. Chi-squared tests were used to compare the proportion of level 4.0 nonachievement under both systems. RESULTS: Milestones 1.0 data consisted of 13,866 residents and Milestones 2.0 data consisted of 7,633 residents. For 1.0 and 2.0, all competency domain median ratings were higher for subsequent years of training. Milestones 2.0 had significantly higher median ratings at all PGY levels for all competency domains except Medical Knowledge. Percentages of PGY-5 residents not achieving the graduation target in Milestones 1.0 ranged from 27% to 42% and in 2.0 from 5% to 13%. For Milestones 1.0, all subcompetencies showed an increased number of residents achieving the graduation target from 2014 to 2019. CONCLUSIONS: This study of General Surgery Milestones 1.0 and 2.0 data uncovered significant increases in average ratings and significantly fewer residents not achieving the graduation target under the 2.0 system. We hypothesize that these findings may be related more to rating bias given the change in rating scales, rather than a true increase in resident ability.

Accreditation Council for Graduate Medical Education Milestone Training Ratings and Surgeons' Early Outcomes.

Importance: National data on the development of competence during training have been reported using the Accreditation Council for Graduate Medical Education (ACGME) Milestones system. It is now possible to consider longitudinal analyses that link Milestone ratings during training to patient outcomes data of recent graduates. Objective: To evaluate the association of in-training ACGME Milestone ratings in a surgical specialty with subsequent complication rates following a commonly performed operation, endovascular aortic aneurysm repair (EVAR). Design, Setting, and Participants: This study of patient outcomes followed EVAR in the Vascular Quality Initiative (VQI) registry (4213 admissions from 208 hospitals treated by 327 surgeons). All surgeons included in this study graduated from ACGME-accredited training programs from 2015 through 2019 and had Milestone ratings 6 months prior to graduation. Data were analyzed from December 1, 2021, through September 15, 2023. Because Milestone ratings can vary with program, they were corrected for program effect using a deviation score from the program mean. Exposure: Milestone ratings assigned to individual trainees 6 months prior to graduation, based on judgments of surgical competence. Main Outcomes and Measures: Surgical complications following EVAR for patients treated by recent graduates during the index hospitalization, obtained using the nationwide Society for Vascular Surgery Patient Safety Organization's VQI registry, which includes 929 participating centers in 49 US states. Results: The study included outcomes for 4213 patients (mean [SD] age, 73.25 [8.74] years; 3379 male participants [80.2%]). Postoperative complications included 9.5% major (400 of 4213 cases) and 30.2% minor (1274 of 4213 cases) complications. After adjusting for patient risk factors and site of training, a significant association was identified between individual Milestone ratings of surgical trainees and major complications in early surgical practice in programs with lower mean Milestone ratings (odds ratio, 0.50; 95% CI; 0.27-0.95). Conclusions and Relevance: In this study, Milestone assessments of surgical trainees were associated with subsequent clinical outcomes in their early career. Although these findings represent one surgical specialty, they suggest Milestone ratings can be used in any specialty to identify trainees at risk for future adverse patient outcomes when applying the same theory and methodology. Milestones data should inform data-driven educational interventions and trainee remediation to optimize future patient outcomes.

Let's Chalk About It: Introducing the TinyTalks Curriculum, a Paradigm for Short, Virtual Chalk Talks.

ABSTRACT: In busy clinical environments, educational opportunities must be designed to accommodate learner-specific needs. Many adult learners prefer short, relevant, technology-enhanced learning. As such, electronic learning (e-learning) experiences have become a prominent part of medical education. Yet, there remain challenges to e-learning experiences in the current educational landscape. To address these challenges, the authors developed the TinyTalks paradigm, which serves as the educational foundation for the TinyTalks curriculum.The TinyTalks paradigm was developed using the existing e-learning literature and foundational principles of adult learning and related theories. The TinyTalks paradigm includes 3 ground rules: (1) all TinyTalks videos must identify a category (approach to, explanation of, or application of) to clarify the focus of the topic, (2) all TinyTalks videos must be less than 7 minutes with all material presented on one virtual chalkboard screen, and (3) all TinyTalks videos must use the hook, frame, and delivery model, which guides the creation of the video. The resulting TinyTalks curriculum is an online repository of short, chalk talk-style educational videos, developed by interdisciplinary health professionals and targeted to the level of trainees, that is available to be used flexibly by learners for just-in-time learning, flipped classroom sessions, and/or self-study.The authors used Kern's 6-step approach to curriculum development as the conceptual framework for the development and implementation of the TinyTalks curriculum at Mass General for Children (June 2021-January 2023). While developing and implementing the curriculum, the authors focused on topic selection, stakeholder recruitment, establishing a process flow, and creating a virtual home.The authors believe the TinyTalks paradigm outlines an effective educational strategy that accommodates the unique needs of both learners and teachers in the medical education setting. The next steps are to scale the TinyTalks curriculum up by expanding the content library and to evaluate its efficacy.

Measuring and Predicting Faculty Consensus Rankings of Standardized Letters of Evaluation.

Background Standardized letters of evaluation (SLOE) are becoming more widely incorporated into the residency application process to make the letter of recommendation, an already critical component in a residency application packet, more objective. However, it is not currently known if the reviewers of these letters share consensus regarding the strength of an applicant determined by their SLOE. Objective We measured the level of faculty agreement regarding applicant competitiveness as determined by SLOEs and the ability of 2 algorithms to predict faculty consensus rankings. Methods Using data from the 2021-2022 Match cycle from the Council of Residency Directors in Emergency Medicine SLOE Database as a blueprint, authors created 50 fictional SLOEs representative of the national data. Seven faculty then rated these SLOEs in order of applicant competitiveness, defined as suggested rank position. Consensus was evaluated using cutoffs established a priori, and 2 prediction models, a point-based system and a linear regression model, were tested to determine their ability to predict consensus rankings. Results There was strong faculty consensus regarding the interpretation of SLOEs. Within narrow windows of agreement, faculty demonstrated similar ranking patterns with 83% and 93% agreement for "close" and "loose" agreement, respectively. Predictive models yielded a strong correlation with the consensus ranking (point-based system r=0.97, linear regression r=0.97). Conclusions Faculty displayed strong consensus regarding the competitiveness of applicants via SLOEs, adding further support to the use of SLOEs for selection and advising. Two models predicted consensus competitiveness rankings with a high degree of accuracy.

Development and validity evidence for the resident-led large group teaching assessment instrument in the United States: a methodological study.

PURPOSE: Despite educational mandates to assess resident teaching competence, limited instruments with validity evidence exist for this purpose. Existing instruments do not allow faculty to assess resident-led teaching in a large group format or whether teaching was interactive. This study gathers validity evidence on the use of the Resident-led Large Group Teaching Assessment Instrument (Relate), an instrument used by faculty to assess resident teaching competency. Relate comprises 23 behaviors divided into six elements: learning environment, goals and objectives, content of talk, promotion of understanding and retention, session management, and closure. METHODS: Messick's unified validity framework was used for this study. Investigators used video recordings of resident-led teaching from three pediatric residency programs to develop Relate and a rater guidebook. Faculty were trained on instrument use through frame-of-reference training. Resident teaching at all sites was video-recorded during 2018-2019. Two trained faculty raters assessed each video. Descriptive statistics on performance were obtained. Validity evidence sources include: rater training effect (response process), reliability and variability (internal structure), and impact on Milestones assessment (relations to other variables). RESULTS: Forty-eight videos, from 16 residents, were analyzed. Rater training improved inter-rater reliability from 0.04 to 0.64. The Φ-coefficient reliability was 0.50. There was a significant correlation between overall Relate performance and the pediatric teaching Milestone, r = 0.34, P = .019. CONCLUSION: Relate provides validity evidence with sufficient reliability to measure resident-led large-group teaching competence.

Measuring Emergency Department Workload Perception Using Electronic Medical Record Measures of Patient Volume and Acuity.

BACKGROUND: Workload in the emergency department (ED) fluctuates and there is no established model for measurement of clinician-level ED workload. OBJECTIVE: The aim of this study was to measure perceived ED workload and assess the relationship between perceived workload and objective measures of workload from the electronic medical record (EMR). METHODS: This study was conducted at a tertiary care, academic ED from July 1, 2020 through April 13, 2021. Attending workload perceptions were collected using a 5-point scale in three care areas with variable acuity. We collected eight EMR measures thought to correlate with perceived workload. EMR values were compared across areas of the department using ANOVA and correlated with attending workload ratings using linear regression. RESULTS: We collected 315 unique workload ratings, which were normally distributed. For the entire department, there was a weak positive correlation between reported workload perception and mean percentage of inpatient admissions (r = 0.23; p < 0.001), intensive care unit admissions (r = 0.2; p < 0.001), patient arrivals per shift (r = 0.14; p = 0.017), critical care billed visits (r = 0.22; p < 0.001), cardiopulmonary resuscitation code activations (r = 0.2; p < 0.001), and level 5 visits (r = 0.13; p = 0.02). There was weak negative correlation for ED discharges (r = -0.23; p < 0.001). Several correlations were stronger in individual care areas, including percent admissions in the lowest-acuity area (r = 0.43; p = 0.033) and patient arrivals in the highest-acuity area (r = 0.44; p < .01). No significant correlation was found in any area for observation admissions or trauma activations. CONCLUSIONS: In this study, EMR measures of workload were not closely correlated with ED attending physician workload perception. Future study should examine additional factors contributing to physician workload outside of the EMR.

Human factors considerations in distance simulation: A nominal group technique application.

INTRODUCTION: The pandemic-driven surge in global distance simulation (DS) adoption highlighted the need for effective educator training. A literature search identified the gap regarding human factors (HF) considerations for the professional development of DS practitioners. This study addresses this gap by applying HF principles to guide educators in developing and delivering evidence-based DS. METHODS: This was a consensus-gathering, three-phase study using the nominal group technique (NGT) in the first phase, qualitative thematic analysis with member checking in the second phase and external expert opinion in the third. A dichotomised approach was used to divide the post-consensus discussion survey results into an agreement and non-agreement for quantitative analysis. RESULTS: The results of the quantitative analysis identified the following needs: developing a conceptual framework for DS, tailoring the technical aspect to the educational objectives, investigating learner engagement, training faculty at an earlier stage and identifying at-risk students. Qualitative results identified primary themes of technology, people and outcome measurements. Key aspects of technology were identified as system- and programme-fit and resource considerations. Outcome measurement highlights the need for increased measurement and research at all levels of DS. DISCUSSION: Specific HF focal points include human-technology interaction and learning outcome assessment within the DS context. Incorporating HF principles throughout the DS process, from inception to outcome evaluation, promises substantial benefits for both learners and educators. This approach empowers both learners and educators, fostering a dynamic and enriched educational environment and improved learning experiences.

Assessing methodological quality in dental education research using MERSQI: Analysis of publications from two journals.

PURPOSE: The Medical Education Research Study Quality Instrument (MERSQI) has been used frequently to assess the methodological quality of medical education but not for dental education. The present study aimed to assess the methodological quality using MERSQI scores of articles published in the Journal of Dental Education (JDE) and the European Journal of Dental Education (EJDE). METHODS: A cross-sectional assessment of the quality of manuscripts published in 2012, 2017, and 2022 JDE and EJDE was conducted. MERSQI data, numbers of authors, first and corresponding author degrees, geographic origins, and funding information were also extracted for each included study. Descriptive and analytical statistics were conducted, and significance level was set at α < 0.05. RESULTS: Four hundred ninety-five articles met the inclusion criteria. The most common study design was a single-group cross-sectional or single-group posttest and conducted in one institution for all studied years. In all journals and years, studies were assessed mainly by participants. The study outcome was mostly satisfaction, attitudes, perceptions, opinions, and general facts. The total mean MERSQI score for each journal and year varied. Year and geographic origin significantly affected the total MERSQI score. Papers originating from Asia had the highest score, followed by South America, Europe, North America, Oceania, and Africa. CONCLUSION: MERSQI score is applicable to the assessment of the methodological quality of dental educational research. The MERSQI score for most of the domains was similar for both journals. The MERSQI score was affected by publication years and geographic origins.

Prevalence and quality of medical Spanish education in US osteopathic medical schools: a national survey.

CONTEXT: Spanish is the language in the United States with the greatest language-concordant physician deficit. Allopathic medical Spanish programs have proliferated, but the national prevalence of medical Spanish education at osteopathic medical schools has never been evaluated. OBJECTIVES: The objectives of this study are to describe the medical Spanish educational landscape at US osteopathic schools and evaluate program adherence to previously established basic standards. METHODS: Between March and October 2022, surveys were sent to all 44 member schools of the American Association of Colleges of Osteopathic Medicine (AACOM). For nonrespondents, data were obtained from publicly available websites. Primary surveys were sent to deans or diversity, equity, and inclusion officers at each osteopathic school to determine whether medical Spanish was offered and to identify a medical Spanish leader. Medical Spanish leaders received the secondary survey. The main measures of this study were the prevalence of medical Spanish programs at osteopathic schools and the extent to which existing programs met each of the four basic standards: having a faculty educator, providing a curricular structure, assessing learner skills, and awarding institutional course credit. RESULTS: We gathered medical Spanish information from 90.9 % (40/44) of osteopathic schools. Overall, 88.6 % (39/44) offered medical Spanish, of which 66.7 % (26/39) had formal curricula, 43.6 % (17/39) had faculty educators, 17.9 % (7/39) assessed learner skills, and 28.2 % (11/39) provided course credit. Only 12.8 % (5/39) of osteopathic schools with medical Spanish programs met all basic standards. Urban/suburban schools were likelier to offer medical Spanish than rural schools (p=0.020). Osteopathic schools in states with the highest Spanish-speaking populations were more likely to offer student-run initiatives (p=0.027). CONCLUSIONS: Most osteopathic schools provide medical Spanish education, but work is needed to improve consistency, quality, and sustainability. Future research should focus on osteopathic student language proficiency assessment, improve medical Spanish accessibility for students at rural programs, and explore the unique content areas of osteopathic medical Spanish education.

Medical Spanish Graphic Activity: A MeGA Deliberate Practice Approach to Reducing Jargon Use With Spanish-Speaking Acute Care Patients.

Introduction: Medical Spanish courses in US medical schools aim to teach patient-centered communication, yet many existing resources focus on technical vocabulary and may inadvertently increase jargon use with patients. Graphic medicine presents an opportunity for interactive learning that centers the patient experience, yet it has never been explored in medical Spanish education. Methods: We developed a Medical Spanish Graphic Activity (MeGA) for medical student deliberate practice of patient-centered verbal communication focused on three aspects: diagnosis, treatment, and follow-up care. Each 30-minute activity included a comics handout depicting a patient with a common problem. Students used voice-to-text technology to record their explanations in response to prompts. Transcripts were analyzed for jargon use, including total jargon, unexplained jargon, and problem words (non-Spanish words plus unexplained jargon), utilizing a previously published, reliable protocol for Spanish medical jargon classification. Participants voluntarily provided postactivity feedback. Results: Twenty-nine fourth-year students with intermediate or greater Spanish skills participated in a series of 10 MeGA activities between January and April 2022. Unexplained jargon use and problem words progressively decreased for all transcripts (diagnosis, treatment, and follow-up; all ps < .001). Total jargon use also decreased, but this was not significant in follow-up transcripts (p = .38). All students agreed that MeGA helped them enhance communication skills applicable to patient care and self-identify strengths and limitations. Discussion: MeGA is realistic to implement, engages students' active participation in the speaking domain, and reduces unexplained jargon use. Future studies should explore the broader application of this model and engage patient perspectives.

Analysis of narrative assessments of internal medicine resident performance: are there differences associated with gender or race and ethnicity?

BACKGROUND: Equitable assessment is critical in competency-based medical education. This study explores differences in key characteristics of qualitative assessments (i.e., narrative comments or assessment feedback) of internal medicine postgraduate resident performance associated with gender and race and ethnicity. METHODS: Analysis of narrative comments included in faculty assessments of resident performance from six internal medicine residency programs was conducted. Content analysis was used to assess two key characteristics of comments- valence (overall positive or negative orientation) and specificity (detailed nature and actionability of comment) - via a blinded, multi-analyst approach. Differences in comment valence and specificity with gender and race and ethnicity were assessed using multilevel regression, controlling for multiple covariates including quantitative competency ratings. RESULTS: Data included 3,383 evaluations with narrative comments by 597 faculty of 698 residents, including 45% of comments about women residents and 13.2% about residents who identified with race and ethnicities underrepresented in medicine. Most comments were moderately specific and positive. Comments about women residents were more positive (estimate 0.06, p 0.045) but less specific (estimate - 0.07, p 0.002) compared to men. Women residents were more likely to receive non-specific, weakly specific or no comments (adjusted OR 1.29, p 0.012) and less likely to receive highly specific comments (adjusted OR 0.71, p 0.003) or comments with specific examples of things done well or areas for growth (adjusted OR 0.74, p 0.003) than men. Gendered differences in comment specificity and valence were most notable early in training. Comment specificity and valence did not differ with resident race and ethnicity (specificity: estimate 0.03, p 0.32; valence: estimate - 0.05, p 0.26) or faculty gender (specificity: estimate 0.06, p 0.15; valence: estimate 0.02 p 0.54). CONCLUSION: There were significant differences in the specificity and valence of qualitative assessments associated with resident gender with women receiving more praising but less specific and actionable comments. This suggests a lost opportunity for well-rounded assessment feedback to the disadvantage of women.

Promoting Longitudinal and Developmental Computer-Based Assessments of Clinical Reasoning: Validity Evidence for a Clinical Reasoning Mapping Exercise.

PURPOSE: Clinical reasoning is vitally important for practitioners across the health professions. However, the assessment of clinical reasoning remains a significant challenge. Combined with other assessment methods, non-workplace-based assessment can increase opportunities to address multiple components of clinical reasoning, evaluate growth, and foster learning, but tools with validity evidence to assess clinical reasoning outside the workplace are scare. This study examined validity evidence for a novel clinical reasoning mapping exercise (CResME). METHOD: Data include CResME performance scores from 120 third-year medical students at the University of Central Florida for 3 topics in May 2022. Each CResME was scored by 2 physician raters based on a scoring rubric that included a combined diagnosis and sequence score. Descriptive statistics were used to examine trends in scores. The authors gathered validity evidence for response process, internal structure, and relations to other variables. RESULTS: The overall mean (SD) score across cases was 66 (29). Internal consistency reliability of cases (Cronbach α) ranged from 0.75 to 0.91. The Phi and G coefficients were 0.45 and 0.56, respectively. Students accounted for 10% of the total variance, indicating the ability to differentiate high and low clinical reasoning skills; the interaction between learner and case accounted for 8.1% of the variance, demonstrating case specificity. There was a moderate correlation between the overall CResME scores and the mean overall score of patient encounter notes from an objective structured clinical examination performed at the end of the third year (0.46; P = .001). Significant associations were also found between the CResME scores and subject exam scores. CONCLUSIONS: The CResME can be used to facilitate the assessment of clinical reasoning, supporting the developmental progress of learners throughout the curriculum. Future research is needed to gather validity evidence for CResMEs with different learners across different settings and formats.

Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation.

BACKGROUND: While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. OBJECTIVE: To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). DESIGN: Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018-2019. Participating faculty (n = 77) observed standardized resident-patient encounters and subsequently completed rater assessment forms (RAFs). SUBJECTS: Participating faculty in longitudinal rater training. MAIN MEASURES: Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). KEY RESULTS: Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (ß 0.45, CI 0.12-0.78, p 0.01). This persisted despite adjusting for the faculty's entrustment ratings. There were no other significant linguistic differences by gender. CONCLUSIONS: Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident's scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.

The rapid growth and expansion of Master of health professions education (MHPE) programs: A mixed-methods study of international curricular trends and guidelines for programs.

PURPOSE: This study explores international trends and standards of Master's degree programs through a comprehensive environmental scan and focus group interviews to understand curricular structure, content, program director expectations, educational context, and future directions. METHOD: Authors conducted a two-phase mixed-methods sequential explanatory design to conduct the environmental scan (phase 1), and subsequently conducting focus groups (phase 2) with program directors. A population list of Master's programs was used to generate a sampling frame, considering the geographic region (continent) and institution type (university, organization, public institution). Qualitative data were coded to analyze the breadth and depth of courses. Three one-hour virtual focus group interviews were conducted with ten program directors. RESULTS: The population list of 159 Masters programs worldwide was used to create a sample for analysis in the environmental scan (n = 46 Masters programs), representing programs from North America, Europe, Australia, and South Africa. Most programs (39%) delivered their courses online, with 20% exclusively offering an in-person program. Focus group participants indicated expectations of graduates, context in which they learn, as well as future directions for improving health professions education graduate programs. CONCLUSION: Program directors should consider programmatic aims, localized needs, and quality/standard of the program in designing Masters programs, with individualized growth opportunities for learners.

The Association of ACGME Milestones With Performance on American Board of Surgery Assessments: A National Investigation of Surgical Trainees.

OBJECTIVE: To determine the relationship between, and predictive utility of, milestone ratings and subsequent American Board of Surgery (ABS) vascular surgery in-training examination (VSITE), vascular qualifying examination (VQE), and vascular certifying examination (VCE) performance in a national cohort of vascular surgery trainees. BACKGROUND: Specialty board certification is an important indicator of physician competence. However, predicting future board certification examination performance during training continues to be challenging. METHODS: This is a national longitudinal cohort study examining relational and predictive associations between Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings and performance on VSITE, VQE, and VCE for all vascular surgery trainees from 2015 to 2021. Predictive associations between milestone ratings and VSITE were conducted using cross-classified random-effects regression. Cross-classified random-effects logistic regression was used to identify predictive associations between milestone ratings and VQE and VCE. RESULTS: Milestone ratings were obtained for all residents and fellows(n=1,118) from 164 programs during the study period (from July 2015 to June 2021), including 145,959 total trainee assessments. Medical knowledge (MK) and patient care (PC) milestone ratings were strongly predictive of VSITE performance across all postgraduate years (PGYs) of training, with MK ratings demonstrating a slightly stronger predictive association overall (MK coefficient 17.26 to 35.76, ß = 0.15 to 0.23). All core competency ratings were predictive of VSITE performance in PGYs 4 and 5. PGY 5 MK was highly predictive of VQE performance [OR 4.73, (95% CI, 3.87-5.78), P <0.001]. PC subcompetencies were also highly predictive of VQE performance in the final year of training [OR 4.14, (95% CI, 3.17-5.41), P <0.001]. All other competencies were also significantly predictive of first-attempt VQE pass with ORs of 1.53 and higher. PGY 4 ICS ratings [OR 4.0, (95% CI, 3.06-5.21), P <0.001] emerged as the strongest predictor of VCE first-attempt pass. Again, all subcompetency ratings remained significant predictors of first-attempt pass on CE with ORs of 1.48 and higher. CONCLUSIONS: ACGME Milestone ratings are highly predictive of future VSITE performance, and first-attempt pass achievement on VQE and VCE in a national cohort of surgical trainees.

Exploring gender and thematic differences in qualitative assessments of internal medicine resident performance.

INTRODUCTION: Evidence suggests gender disparities in medical education assessment, including differences in ratings of competency and narrative comments provided in resident performance assessments. This study explores how gender manifests within the content of qualitative assessments (i.e., narrative comments or performance feedback) of resident performance. METHODS: Qualitative content analysis was used to explore gender-based differences in narrative comments included in faculty assessments of resident performance during inpatient medicine rotations at six Internal Medicine residency programs, 2016-2017. A blinded, multi-analyst approach was employed to identify themes across comments. Patterns in themes with resident gender and post-graduate year (PGY) were explored, focusing on PGY2 and PGY3 when residents are serving in the team leader role. RESULTS: Data included 3,383 evaluations with narrative comments of 385 men (55.2%) and 313 women residents (44.8%). There were thematic differences in narrative comments received by men and women residents and how these themes manifested within comments changed with training time. Compared to men, comments about women had a persistent relationship-orientation and emphasized confidence over training including as interns and in PGY2 and PGY3, when serving as team leader. The relationship-orientation was characterized not only by the residents' communal attributes but also their interpersonal and communication skills, including efforts supporting others and establishing the tone for the team. Comments about women residents often highlighted confidence, including recommendations around behaviors that convey confidence in decision-making and team leadership. DISCUSSION: There were gender-based thematic differences in qualitative assessments. Comments about women resident team leaders highlight relationship building skills and urge confidence and actions that convey confidence as team leader. Persistent attention to communal skills suggests gendered expectations for women resident team leaders and a lost opportunity for well-rounded feedback to the disadvantage of women residents. These findings may inform interventions to promote equitable assessment, such as providing feedback across the competencies.